Multi-objective optimization control method and system for cooperative ramp merging of connected vehicles on highway

ABSTRACT

State data of vehicles in a control area is collected, then time of starting control and selection of controlled vehicles are decided according to a position of a ramp merging vehicle, information is transmitted into a traffic control module by means of a data transmission module, and an artificial intelligence based ramp merging multi-objective control model ensures efficient, safe, and energy-saving operation of overall traffic of a road while completing ramp merging by means of a vehicle traveling track in the cooperative ramp control area. Compared with a traditional method, the present invention greatly promotes merging of the ramp vehicles, and different from other methods having the defect of transforming the problem of ramp merging into the problem of vehicle sequencing, the present invention greatly improves efficiency of ramp merging,

CROSS REFERENCES

This application is the continuation application of International Application No. PCT/CN2022/102755 filed on 30 Jun. 2022 which designated the U.S. and claims priority to Chinese Application No. CN202210164445.8 filed on 23 Feb. 2022, the entire contents of each of which are hereby incorporated by reference.

TECHNICAL FIELD

The present invention belongs to the technical field of intelligent traffic vehicle-road cooperation, and in particular to a multi-objective optimization control method and system for cooperative ramp merging of connected vehicles on a highway.

BACKGROUND ART

With rapid growth of national economy and continuous promotion of urbanization, increased regional trades have prompted rapid growth in urban trips, thereby bringing challenges to efficient and safe operation of a highway. An on-ramp area, serving as a bottleneck affecting traffic efficiency of the highway, is a particularly eye-catching problem in traffic management. A common ramp control method is to control the rate of on-ramp inflow by adjusting macroscopic traffic flow state variables (such as density and flow). However, a ramp metering strategy aims to prioritize the efficiency of a trunk line and attempt to prevent congestion. With an incapability to control movement of a single vehicle to promote a merging process, the method fail to successfully merge in a case of a number of vehicles on a main line, resulting in congestion and safety problems.

In consideration of excellent dynamic control capabilities of intelligent connected vehicles, a cooperative ramp merging strategy was provided, to better control merging by adjusting tracks of the vehicles, so as to ensure traffic efficiency and safety. Common merging strategies can be divided into heuristic methods (rule-based methods or fuzzy methods) and optimal methods. However, heuristic algorithms usually require domain-specific expertise to finely design certain driving rules, lack of adaptability to other unknown situations, and have difficulty in achieving optimal control. Optimization algorithms such as a Pontryagin’s maximum principle (PMP) and dynamic programming (DP) lack of self-learning capabilities, have difficulty in achieving global optimization in a complex on-ramp merging environment, and lack of adaptability to problems, and although reinforcement learning based optimization algorithms can improve adaptability of a system to unknown environments, existing strategies are too focused on reward shaping and model construction to fundamentally solve the problem of cooperative ramp merging. In essence, it is necessary for cooperative ramp merging to satisfy multi-objective long-term and short-term feedback at the same time. That is, it is necessary to consider the influence on long-term objectives during execution of short-term actions, but rewards for on-ramp merging evaluation are sparse and long-term for the merging problem. In this case, a traditional reinforcement learning method has difficulty in avoiding local optimality and instability when dealing with sparse reward problem, thereby influencing traffic efficiency and safety of vehicles in a ramp area.

Aiming at the above problem, the inventors have provided a related paper for a battery life oriented on-ramp reinforcement learning method, which initially solves the problem of on-ramp merging. However, the method is mainly oriented to battery health, is an implementation case in a specific application scenario of a new energy vehicle, but is not applicable to a real traditional mixed ramp merging scenario of a fuel vehicle and the new energy vehicle; and moreover, the paper does not involve ramp merging vehicle selection and main road merging gap selection, and lacks of key links in actual ramp merging application. Aiming at the above problem, the present patent, on the basis of specific cases provided by the paper, provided a multi-objective optimization control method and system for cooperative ramp merging of connected vehicles on a highway, which establish a comprehensive and complete framework and an advanced technology methodology for the problem of on-ramp vehicle merging.

SUMMARY

In order to solve the above technical problem, the present invention provides a multi-objective optimization control method and system for cooperative ramp merging of connected vehicles on a highway, which ensure efficient, safe, and energy-saving operation of overall traffic of a road while completing ramp merging by means of a vehicle traveling track in a cooperative ramp control area.

The present invention provides a multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway. The control method includes:

-   step 1, collecting state data of vehicles in a control area of the     highway, and analyzing and processing the state data, the control     area including an intersection of a main road and a ramp, a merging     area, and partial road segments of the main road, the ramp and an     acceleration lane of the highway, a range of the control area being     a communication range of a road side unit, the road side unit being     arranged at the intersection of the main road and the ramp of the     highway, and the merging area being a preselected area including the     partial road segments of the acceleration lane and road segments of     the main road parallel to the partial road segments of the     acceleration lane; -   step 2, constructing a set of alternatives for a ramp merging     vehicle, an auxiliary vehicle, and a guidance vehicle according to     the state data of the vehicles in the control area; -   step 3, inputting the set of alternatives into an artificial     intelligence based ramp merging multi-objective control model     separately, and further determining selection of the auxiliary     vehicle and the guidance vehicle by means of an optimal value     strategy; -   step 4, according to the auxiliary vehicle, the guidance vehicle,     and the merging vehicle that are selected, controlling and adjusting     accelerations of the auxiliary vehicle and the guidance vehicle, to     ensure safe merging of the ramp merging vehicle from the     acceleration lane into the main road in the selected merging area;     and -   step 5, collecting state data of the ramp merging vehicle and the     auxiliary vehicle, of which the accelerations are adjusted, and     returning to step 4 to adjust accelerations at a next moment.

As a further improvement of the control method of the present invention, in step 1, the state data of the vehicles includes positions, speeds, power battery states, and corresponding moment information of the vehicles in the control area.

As a further improvement of the control method of the present invention, in step 1, the analyzing and processing the state data includes: analyzing the data, extracting features, and fusing information.

As a further improvement of the control method of the present invention, in steps 2 and 3, the optimal value strategy selected for the ramp merging vehicle, the guidance vehicle, and the auxiliary vehicle is as follows:

-   1)selecting the merging vehicle: setting a vehicle, of which a front     bumper is closest to a ramp exit line, on the ramp as the merging     vehicle, and obtaining state information of all the vehicles in the     control area in first t time steps; -   2)according to position information of all the vehicles, preliminary     selecting each z main road vehicles in the rear of and in front of     the merging vehicle as alternative vehicles of the auxiliary vehicle     and the guidance vehicle on the basis of front and rear relations     between main line vehicles and the merging vehicle; -   3)in all the alternative vehicles, selecting two vehicles that are     adjacent to each other as a group of guidance vehicle and auxiliary     vehicle, to construct an alternative set AL of the guidance vehicle     and the auxiliary vehicle of the merging vehicle; and -   4)using a traversal method to substitute combinations in the     alternative set AL into the artificial intelligence based ramp     merging multi-objective control model separately, and determining     the merging vehicle, the auxiliary vehicle, and the guidance vehicle     that are finally selected on the basis of a value function Q_(π) of     the model.

As a further improvement of the control method of the present invention, the state information of all the vehicles in the control area in the first time steps includes speeds, positions, and accelerations.

As a further improvement of the control method of the present invention, in step 4) of the optimal value strategy selected for the merging vehicle, the auxiliary vehicle, and the guidance vehicle, constructing an objective function and constraints of the artificial intelligence based ramp merging multi-objective control model successfully merged into a moment includes:

-   1) setting τ as a safe merging moment to construct a position and     speed relation to be satisfied by the guidance vehicle, the     auxiliary vehicle, and the ramp merging vehicle at the safe merging     moment:

-   $\begin{matrix}     {x_{m}\left( \hat{t} \right) \leq x_{l}\left( \hat{t} \right) - \left( {L_{1} + s_{0}} \right) - \tau v_{m}\left( \hat{t} \right)} & \text{­­­(1)}     \end{matrix}$

-   $\begin{matrix}     {x_{m}\left( \hat{t} \right) \geq x_{f}\left( \hat{t} \right) + \left( {L_{1} + s_{0}} \right) - \tau v_{f}\left( \hat{t} \right)} & \text{­­­(2)}     \end{matrix}$

-   $\begin{matrix}     {v_{m}\left( \hat{t} \right) \leq v_{l}\left( \hat{t} \right)} & \text{­­­(3)}     \end{matrix}$

-   $\begin{matrix}     {v_{m}\left( \hat{t} \right) \geq v_{f}\left( \hat{t} \right)} & \text{­­­(4)}     \end{matrix}$

-   $\begin{matrix}     {x_{m}\left( \hat{t} \right) \in \left\lbrack {d_{min},d_{max}} \right\rbrack} & \text{­­­(5)}     \end{matrix}$

-   -   x_(l), v_(l) and a_(l)representing a position, a speed, and an         acceleration of the guidance vehicle;     -   x_(f), v_(f)and a_(f) representing a position, a speed, and an         acceleration of the auxiliary vehicle;     -   x_(m), v_(m)and a_(m) representing a position, a speed, and an         acceleration of the ramp merging vehicle; τ being a constant         time interval, L₁ being a length of the vehicle, s₀ being a         pause gap, d_(min) and d_(max)being a start point and an end         point of the merging area respectively, and the merging area         having a length of d_(max) - d_(min;) and the equation         sequentially representing, from top to bottom, that the ramp         merging vehicle is at the rear of the guidance vehicle, the ramp         merging vehicle is in front of the auxiliary vehicle, the ramp         merging vehicle and the guidance vehicle have a consistent         speed, the ramp merging vehicle and the auxiliary vehicle have a         consistent speed, and the ramp merging vehicle is safely merged         from the acceleration lane into the main road in the selected         merging area; and

-   2) under the condition of satisfying 1), further constructing an     objective function C including, but not limited to, objectives of     driving comfort, vehicle energy consumption, traffic efficiency,     etc;

-   $\begin{matrix}     {minC\left( \hat{t} \right) = min\mspace{6mu}{\int_{0}^{\hat{t}}\left( {c^{1}C_{1}\left( \hat{t} \right) + c^{2}C_{2}\left( \hat{t} \right) + \cdots + c^{n}C_{n}\left( \hat{t} \right)} \right)}} & \text{­­­(6)}     \end{matrix}$

-   C_(n)(t̂) representing cost functions of different objectives, and     c^(n) representing parameters.

As a further improvement of the control method of the present invention, in step 4) of the optimal value strategy selected for the merging vehicle, the guidance vehicle and the auxiliary vehicle, the artificial intelligence based ramp merging multi-objective control model is solved by using a reinforcement learning actor-critic algorithm, which specifically includes:

-   1) establishing a state space -   𝒮 -   and a behavior space -   𝒜 -   selecting, according to the state data of the guidance vehicle, the     auxiliary vehicle, and the ramp merging vehicle, six-dimensional     state information s = {x_(l), x_(m), x_(f), v_(l), v_(m), v_(f)} to     represent the most relevant influence factor in an environment, s ∈ -   𝒮 -   , and selecting a control behavior strategy a = {a_(m), a_(f)}, a ∈ -   𝒜 -   on the basis of a control object; -   2) establishing an optimal objective: according to a vehicle     constraint relation at a safe merging moment t̂, constructing an     optimal objective space set -   g* = {g₁^(*), g₂^(*), g₃^(*), g₄^(*), g₅^(*)} -   for ramp merging, -   g* ∈ 𝒢, 𝒢 -   being an objective space set, g₁ representing that equation (1) that     the merging vehicle is at the rear of the guidance vehicle is     satisfied, -   g₂^(*) -   representing that equation (2) that the ramp merging vehicle is in     front of the auxiliary vehicle is satisfied, -   g₃^(*) -   representing that equation (3) that the ramp merging vehicle and the     guidance vehicle have a consistent speed is satisfied, -   g₄^(*) -   representing that equation (4) that the ramp merging vehicle and the     auxiliary vehicle have a consistent speed is satisfied, and -   g₅^(*) -   representing that equation (5) that the ramp merging vehicle is     safely merged from the acceleration lane into the main road in the     selected merging area is satisfied; -   3) constructing an objective space: on the basis of classifications     included by the optimal objective space set -   g* = {g₁^(*), g₂^(*), g₃^(*), g₄^(*), g₅^(*)} -   in 2), establishing the objective space set -   𝒢 -   to satisfy g = (g₁,g₂, g₃,g₄, g₅}, g₁, representing a position     relation between the ramp merging vehicle and the guidance vehicle,     g₂ representing a position relation between the ramp merging vehicle     and the auxiliary vehicle, g₃ representing a speed relation between     the ramp merging vehicle and the guidance vehicle, g₄ representing a     speed relation between the ramp merging vehicle and the auxiliary     vehicle, and g₅ representing a relation between a position of the     ramp merging vehicle and the merging area; -   4) constructing rewards: with a reward function being -   r : 𝒮 × 𝒜 × 𝒢 → R -   , introducing, according to safe, efficient and comfortable     traveling requirements, at least two short-term objective rewards     under the condition that the reward r(s, a, g*) at each time step     includes a long-term objective merging reward R_(m)(t), the     long-term objective merging reward R_(m)(t) that must be inc3luded     being represented as follows: -   $\begin{matrix}     \begin{array}{l}     {R_{m}(t) =} \\     \left\{ \begin{array}{l}     {\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu} 1,\mspace{6mu}\mspace{6mu}\mspace{6mu}\text{safe successful merging}} \\     {0,\mspace{6mu}\mspace{6mu}\mspace{6mu}\text{the merging vehicle can}\left( \text{may} \right)\text{be merged, but}g\text{*}\mspace{6mu} is\mspace{6mu} not\mspace{6mu} satisfied} \\     {\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu} - 1\mspace{6mu},\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\text{others}}     \end{array} \right)     \end{array} & \text{­­­(7)}     \end{matrix}$ -   5) obtaining a data link: according to state, objective, strategy     and reward data of the t^(th) time step obtained from 1) to 4),     obtaining the data link (s_(t)||g*,a,r,s_(t+1)||g*), and storing the     data into an intelligent optimization module, s||g* representing a     connection between a state s and an objective g*; -   6) expanding data exploration, and optimizing the objective space:     further providing a multi-experience replay based virtual objective     construction algorithm, to introduce virtual objectives, to optimize     the objective space, and simultaneously expand the data exploration;     and -   7) in each time step, according to the data link stored by the     intelligent optimization module, on the basis of an actor-critic     algorithm framework, training a merging control strategy by means of     a deep neural network taking θ_(A) as a parameter, the strategy     directly outputting an action to control acceleration magnitudes and     states of the ramp merging vehicle and the auxiliary vehicle and     objective inputs, strategy optimization aiming at to find an optimal     behavior strategy a, to maximize a return expectation of a whole     trip, and finally, the optimal control strategy being output by     means of forward delivery of a trained network: a = π(s,g | θ_(A)).

As a further improvement of the control method of the present invention, the short-term objective rewards in construction of the rewards of the reinforcement learning actor-critic algorithm includes, but not limited to, an energy-saving reward R_(e)(t), a comfort reward R_(p)(t), an efficient traffic reward R_(s)(t), and a battery state reward R_(b)(t), and

a coupled Chebyshev based multi-objective reward optimization method for the reinforcement learning actor-critic algorithm includes:

-   (1) determining optimized reward terms: under the condition of     assuming that the number of a variety of introduced real-time     short-term objectives after successful merging is n_(r), determining     the number of the optimized reward terms for a merging problem as     n_(r); -   (2) determining a super-ideal optimal value of each reward:     constructing the super-ideal optimal value -   r_(i)^(**) = r_(i)^(*) + ϑ_(i)  (i=1, 2,  …,)  n_(r) -   of each objective, -   r_(i)^(*) -   being an ideal value, and ϑ_(i) being a constant according to     selection of empirical data, and representing a degree that the     super-ideal optimal value is better than the ideal value; -   (3) constructing a generalized weighted Chebyshev optimization model     for a multi-objective problem: with setting λ_(i) as a Chebyshev     weight of a short-term objective, transforming the multi-objective     problem into a generalized weighted Chebyshev multi-objective     problem as follows: -   $\begin{matrix}     {\min\left\{ {F_{c} + \rho{\sum_{i = 1}^{n_{r}}\left( {r_{i}^{\ast \ast} - r_{i}} \right)}} \right\}} & \text{­­­(14)}     \end{matrix}$

The following condition being satisfied:

$\begin{matrix} \begin{array}{l} {F_{c} - \lambda_{i}\left( {r_{i}^{\ast \ast} - r_{i}} \right) \geq 0\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu} 1 \leq i \leq n_{r}} \\ {r_{i}\mspace{6mu}\text{being an unrestricted variable}\mspace{6mu}\mspace{6mu}\text{1} \leq i \leq n_{r}} \\ {F_{c} \geq 0} \end{array} & \text{­­­(15)} \end{matrix}$

F_(c)being a weighted Chebyshev norm,

F_(c) = max {λ_(i)(r_(i)^(**) − r_(i))}

representing a maximum deviation between each objective and the super-ideal optimal value; and

$\rho{\sum_{i = 1}^{n_{r}}\left( {r_{i}^{\ast \ast} - r_{i}} \right)}$

being a term ensuring the stable algorithm, usually, ρ = 0.001, and a computation equation of λ_(i) being referred to as follows:

$\begin{matrix} {\lambda_{i} = \left\{ \begin{array}{ll} {\frac{1}{\left( {r_{i}^{\ast \ast} - r_{i}} \right)}\left\lbrack {\sum_{i = 1}^{n_{r}}\frac{1}{\left( {r_{i}^{\ast \ast} - r_{i}} \right)}} \right\rbrack^{- 1}} & {r_{i} \neq r_{i}^{\ast \ast}} \\ 1 & {r_{i} = r_{i}^{\ast \ast}} \\ 0 & {r_{i} \neq r_{i}^{\ast \ast},\text{­­­(16)}\mspace{6mu}\text{but}\exists i,\mspace{6mu}\mspace{6mu}\text{and}r_{i} = r_{i}^{\ast \ast}} \end{array} \right)} &  \end{matrix}$

(4) constructing a reward function as:

$\begin{matrix} \begin{array}{l} {R_{m}(t) =} \\ \left\{ \begin{array}{l} {\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu} 1,\mspace{6mu}\mspace{6mu}\mspace{6mu}\text{safe successful merging}} \\ {0,\mspace{6mu}\mspace{6mu}\mspace{6mu}\text{the merging vehicle can}\left( \text{may} \right)\text{be merged, but}g\text{*}\mspace{6mu} is\mspace{6mu} not\mspace{6mu} satisfied.} \\ {\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu} - 1\mspace{6mu},\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\text{others}} \end{array} \right) \end{array} & \text{­­­(17)} \end{matrix}$

As a further improvement of the control method of the present invention, the multi-experience replay based virtual objective construction algorithm provided in construction of the rewards of the reinforcement learning actor-critic algorithm includes:

-   (1) constructing multi-experience virtual objectives and optimizing     the objective space at a t time step: on the basis of the objective     g* and the data link (st||g*, a, r, s_(t+1) ||g*) at the t time     step, constructing a fully connected neural network H, g = H_(φ)(s,     g*)taking φ as a parameter, to obtain 1 virtual objectives in a     current state: g¹ = Hφ ( s_(t), g*), g²=H_(φ)(S_(t+1), g*), ...,     g^(l)=H_(φ)(S_(t+l-1), g*);

-   (2) optimizing data exploration on the basis of the optimized     objective space: constructing 1 virtual objective data links on the     basis of the obtained 1 virtual objectives:

-   $\begin{array}{l}     {\left( {s_{t}\left\| {g^{1},a,r,s_{t + 1}} \right\| g^{1}} \right),\mspace{6mu}} \\     {\mspace{6mu}\left( {s_{t}\left\| {g^{2},a,r,s_{t + 1}} \right\| g^{2}} \right),\mspace{6mu}\mspace{6mu}\ldots,\mspace{6mu}\mspace{6mu}\left( {s_{t}\left\| {g^{l},a,r,s_{t + 1}} \right\| g^{l}} \right);\mspace{6mu}\text{and}}     \end{array}$

-   -   storing the virtual objective data links into the intelligent         optimization module;

-   (3) optimizing data exploration at a full time step: repeating (1)     and (2) for each time step to complete data exploration and     objective space optimization at all time steps;

-   (4) selecting virtual objective effectiveness on the basis of an     artificial intelligence model:     -   training the artificial intelligence model according to an         optimized data set, and selecting,     -   according to a training result, an optimal virtual objective         data link at each time step; and

-   (5) verifying the fully connected neural network H for the virtual     objectives: according to the optimal virtual objective data link at     each time step, verifying the parameter φ of the fully connected     neural network H for the virtual objectives, to constantly improve     accuracy of generation of the virtual objectives, so as to ensure     algorithm performance and a training speed.

The present invention provides a system based on a multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway. The system includes an information collection module, a data transmission module, a traffic control module, and an intelligent optimization module, where

-   the information collection module is used for collecting state data     of vehicles in a control area of a highway, analyzing and processing     the state data, and selecting a ramp merging vehicle, an auxiliary     vehicle, and a guidance vehicle; -   the information collection module includes an in-vehicle unit, and a     road side unit, the road side unit being arranged at an intersection     of a main road and a ramp of a highway, the road side unit being     used for collecting positions, speeds and corresponding moment     information of the vehicles in the control area, and being further     used for collecting time of determining the ramp merging vehicle,     and time when a front bumper reaches a ramp exit line, and the     in-vehicle unit being used for collecting power battery states and     corresponding moment information of the vehicles in the control     area; -   the data transmission module is used for using a mobile     communication technology as a subject information transmission     communication mode to assist one or two of wireless communication     modes of wireless fidelity/bluetooth (WiFi/BT), and dedicated short     range communications (DSRC) to transmit data between the information     collection module and the traffic control module, the traffic     control module and the intelligent optimization module, -   the traffic control module is used for obtaining, according to the     state data of the vehicles provided by the information collection     module, a behavior strategy a, an objective g and a reward r that     are optimal in real time, sending the behavior strategy to the     in-vehicle unit for real-time control of the vehicles, and further     sending the behavior strategy a, the objective g and the reward r     that are optimal to the intelligent optimization module; and -   the intelligent optimization module is used for storing data sent by     the traffic control module, optimizing a ramp merging     multi-objective control model on the basis of an alternative set AL     of the vehicles selected in step 3) and an optimization algorithm     provided in step 4), and transmitting the optimized model to the     traffic control module.

Beneficial Effects

The present invention uses the above technical solution, and has the following technical effects compared with the prior art:

1. The present invention promotes, by controlling the vehicles on two lanes of the ramp merging area, a process of ramp merging, solves the problem of queuing caused by ramp vehicle merging delay caused by a traditional ramp control means only using flow rate control, and further actively promotes merging of the ramp vehicles on the premise of ensuring traffic capability of a main line, thereby greatly improving traffic efficiency in the ramp area; and a single-vehicle control method also ensures safe travel between the vehicles.

2. The present invention uses reinforcement learning for single-vehicle-controlled ramp merging, which is different from other single-vehicle-controlled ramp merging methods. Reinforcement learning may constantly explore and optimize the control strategy from historical data without constructing complex models, and moreover, the algorithm further has some robustness and adaptability due to diversity of exploration.

3. The present invention introduces a multi-experience virtual objective construction method, and the steps of objective space optimization and data exploration, thereby greatly improving data exploration efficiency, effectively solving the problems of long-term reward sparsity and a coupled relation between long rewards and short rewards in multi-objective ramp control, and ensuring safe and effective multi-objective ramp merging.

4. Compared with a paper published by the inventors, the present invention is mainly improved as follows:

1) Different from a battery health oriented merging method only for a new energy vehicle in the paper, the present invention provides a merging method for a mixed ramp scenario applicable to a variety of vehicles (including, but not limited to, traditional fuel vehicles, new energy vehicles, etc.), and the present invention provides a key technical link applied to a real merging scenario of the highway.

2) The present invention further optimizes vehicle constraints at the safe and successful merging moment, and further enriches a selection algorithm for the guidance vehicle, the auxiliary vehicle and the merging vehicle in the control area, and compared with a simple selection mode in the paper, the selection solution of the present invention may obtain the optimal vehicle selection solution, thereby further improving an optimal solution of the algorithm, and making running of the vehicles more energy-saving, efficient and stable.

3) The present invention further optimizes an importance level of the long-term rewards and the short-term rewards, and reserves an interface for objective introduction of the present invention for different practical situations, and the present invention determines a coefficient relation between the short-term rewards, thereby ensuring that the present invention may be adapted to situations having a variety of different objectives, and further improving the scope of application of the present invention.

4) The present invention further optimizes a data exploration algorithm provided by the article, introduces a multi-experience virtual objective construction method, which is different from simple selection of a next state by the paper, further improves efficiency of data utilization, improves a direction and correctness of data exploration, and greatly avoids occurrence of dangerous situations such as collision.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a merging scenario of a highway provided in the present invention; and

FIG. 2 is an architecture diagram of a multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The present invention will be further described in detail below in combination with the accompanying drawings and particular embodiments.

A method provided in the present invention is based on assumptions as follows: 1) network connected vehicles need to have necessary capabilities to transmitting information and implement instructions, that is, the network connected vehicles have the capability to interact information by means of an in-vehicle unit, and the vehicles fully execute control instructions; 2) a ramp control area includes an intersection of a main road and a ramp, a ramp merging area and partial road segments of a main line and the ramp, and a control range of the control area is a communication range of a road side unit; 3) the merging area has a fixed length, i.e., d_(m) = d_(max) - d_(min;) 4) delays in information transmission, data processing and computation, and instruction execution are ignored, that is, it is assumed that a running speed of each module is enough to support running of a system; and 5) influences of lateral movement and temperatures of the vehicles on the vehicles are ignored.

A multi-objective optimization control system for cooperative ramp merging of connected vehicles on a highway provided in the present invention includes an information collection module, a data transmission module, a traffic control module, and an intelligent optimization module.

1) The information collection module collects, by means of an in-vehicle unit and a road side unit, state information such as speeds, positions, and power battery states and corresponding moment information of vehicles in a control area and start moment information of merging of a ramp vehicle (a selected moment of a merging vehicle), and analyzing and processing the data, and the analyzing and processing the data include, but not limited to, analyze the data, extract features, fuse information, etc.; and

information collection cooperation of the in-vehicle unit and the road side unit included in the information collection module is as follows:

-   (1) the road side unit is responsible for collecting traffic state     information, such as the speeds and positions of the vehicles in the     area, and the start moment of merging of the ramp merging vehicle;     and -   (2) the in-vehicle unit is responsible for collecting vehicle     related parameter information, such as engine state information,     battery state information, and gear information. A power composition     of a network connected vehicle designed in the present invention     includes: a fuel vehicle, a pure electric vehicle, a hydrogen energy     vehicle, and a hybrid electric vehicle.

2) The data transmission module uses a fifth generation mobile communication technology (5G) as a subject information transmission communication mode to assist one or two of wireless communication modes of wireless fidelity/bluetooth (WiFi/BT), dedicated short range communications (DSRC), etc. to transmit data between the in-vehicle unit, the road side unit, and each module;

-   3) the traffic control module obtains, according to traffic state     information provided by the information collection module, a     behavior strategy a, an objective g and a reward r that are optimal     in real time, sends the strategy to the in-vehicle unit for     real-time control of the vehicles, and further packages a traffic     state information set to be sent to the intelligent optimization     module; and -   4) the intelligent optimization module includes a training     sub-module and a data sub-module. The data sub-module stores data     sent by the traffic control module, and applies an artificial     intelligence data exploration method to obtain a number of data     pairs for storage; and the training sub-module utilizes a reward     function to optimize an artificial intelligence based ramp merging     multi-objective control model according to the data provided by the     data sub-module, and transmits an optimized model to the traffic     control module.

The control area includes an intersection of a main road and a ramp, a ramp merging area, and partial road segments of a main line and the ramp. It is worth noting that a control range of the control area is a communication range of the road side unit, which sees FIG. 1 for details.

As shown in FIG. 2 , a multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway provided in the present invention includes:

-   S01 obtain, by an information collection module, state data of     vehicles in a control area; -   S02 perform, by the information collection module, data analysis,     information fusion, etc. on the collected data by means of an     intelligent data analysis model; -   S03 according to the state data of the vehicles in the control area,     construct a set AL of alternatives of a ramp merging vehicle, an     auxiliary vehicle and a guidance vehicle; -   S04 input the set of alternatives into an artificial intelligence     based ramp merging multi-objective control model separately, and     further determine selection of the auxiliary vehicle and the     guidance vehicle by means of an optimal value strategy; -   S05 transmit, by a transmission module, the vehicles selected in S04     and vehicle related collected data to a control module, -   S06 confirm, by a traffic control module, the vehicles to which     instructions are sent to implement a decision on the basis of the     artificial intelligence based ramp merging multi-objective control     model, a frame of the artificial intelligence based ramp merging     multi-objective control model being a reinforcement learning     actor-critic algorithm; -   S07 transmit, by the transmission module, decision information to     in-vehicle units of controlled vehicles to achieve a vehicle safe,     efficient and energy-saving ramp merging objective, and the     controlled vehicles are the ramp merging vehicle and the auxiliary     vehicle; and -   S08 collect state information data pairs of the controlled vehicles     under a control strategy, and store the state information data pairs     of the controlled vehicles into a data sub-module to form a closed     loop control.

In S03 and S04, a selection method for the merging vehicle, the auxiliary vehicle and the guidance vehicle is based on a framework of the optimal value strategy algorithm framework, and includes:

-   1) select the merging vehicle: set a vehicle, of which a front     bumper is closest to a ramp exit line, on the ramp as the merging     vehicle, and obtain state information, such as speeds, positions,     and accelerations, of all the vehicles in the control area in first     t time steps; -   2) according to position information of all the vehicles,     preliminary select each z main road vehicles in the rear of and in     front of the merging vehicle as alternative vehicles of the     auxiliary vehicle and the guidance vehicle on the basis of front and     rear relations between main line vehicles and the merging vehicle; -   3) in all the alternative vehicles, select two vehicles that are     adjacent to each other as a group of guidance vehicle and auxiliary     vehicle, to construct an alternative set AL of the guidance vehicle     and the auxiliary vehicle of the merging vehicle; -   4) use a traversal method to substitute combinations in the     alternative set AL into the artificial intelligence based ramp     merging multi-objective control model separately, and determine the     merging vehicle, the auxiliary vehicle, and the guidance vehicle     that are finally selected on the basis of a value function Q″ of the     model; and -   5) control and adjust, by the artificial intelligence based ramp     merging multi-objective control model, accelerations of the merging     vehicle and auxiliary vehicle according to the selected guidance     vehicle, to solve multi-objective optimal control, so as to     successfully merge the merging vehicle into a main line, and     simultaneously ensuring energy consumption optimization of the     merging vehicle and efficient and safe traffic of a road.     Specifically, constructing a problem mathematical model and carrying     out optimization solution on the basis of reinforcement learning are     specifically included.

The constructing a problem mathematical model includes:

-   1) set a safe merging moment as t to construct a position and speed     relation to be satisfied by the guidance vehicle, the auxiliary     vehicle, and the ramp merging vehicle at the safe merging moment:

-   $\begin{matrix}     {x_{m}\left( \hat{t} \right) \leq x_{l}\left( \hat{t} \right) - \left( {L_{1} + s_{0}} \right) - \tau v_{m}\left( \hat{t} \right)} & \text{­­­(1)}     \end{matrix}$

-   $\begin{matrix}     {x_{m}\left( \hat{t} \right) \geq x_{f}\left( \hat{t} \right) + \left( {L_{1} + s_{0}} \right) - \tau v_{f}\left( \hat{t} \right)} & \text{­­­(2)}     \end{matrix}$

-   $\begin{matrix}     {v_{m}\left( \hat{t} \right) \leq v_{l}\left( \hat{t} \right)} & \text{­­­(3)}     \end{matrix}$

-   $\begin{matrix}     {v_{m}\left( \hat{t} \right) \geq v_{f}\left( \hat{t} \right)} & \text{­­­(4)}     \end{matrix}$

-   $\begin{matrix}     {x_{m}\left( \hat{t} \right)\mspace{6mu} \in \left\lbrack {d_{min},d_{max}} \right\rbrack} & \text{­­­(5)}     \end{matrix}$

-   -   x_(l), v_(l) and a_(l)representing a position, a speed, and an         acceleration of the guidance vehicle;     -   x_(f), v_(f) and a_(f) representing a position, a speed, and an         acceleration of the auxiliary vehicle; x_(m), v_(m) and a_(m)         representing a position, a speed, and an acceleration of the         ramp merging vehicle; τ being a constant time interval, L1 being         a length of the vehicle, s₀ being a pause gap, d_(min) and         d_(max)being a start point and an end point of the merging area         respectively, and the merging area having a length ofd_(max) -         d_(min;) and the equation sequentially representing, from top to         bottom, that the ramp merging vehicle is at the rear of the         guidance vehicle, the ramp merging vehicle is in front of the         auxiliary vehicle, the ramp merging vehicle and the guidance         vehicle have a consistent speed, the ramp merging vehicle and         the auxiliary vehicle have a consistent speed, and the ramp         merging vehicle is safely merged from the acceleration lane into         the main road in the selected merging area; and

-   2) under the condition of satisfying 1), further construct an     objective function C including, but not limited to, objectives of     driving comfort, vehicle energy consumption, traffic efficiency,     etc.;

-   $\begin{matrix}     {minC\left( \hat{t} \right) = min\mspace{6mu}{\int_{0}^{\hat{t}}\left( {c^{1}C_{1}\left( \hat{t} \right) + c^{2}C_{2}\left( \hat{t} \right) + \cdots + c^{n}C_{n}\left( \hat{t} \right)} \right)}} & \text{­­­(6)}     \end{matrix}$

-   C_(n)(t̂) representing cost functions of different objectives, and     c^(n) representing parameters.

As the preferred solution of the method of the present invention, the artificial intelligence based ramp merging multi-objective control model is solved by using a reinforcement learning actor-critic algorithm, which specifically includes:

1) establish a state space

S

and a behavior space

A

: selecting, according to the state data of the guidance vehicle, the auxiliary vehicle, and the ramp merging vehicle, six-dimensional state information s = {x_(l), x_(m), x_(f), v_(l) v_(m), v_(f)} to represent the most relevant influence factor in an environment,

s ∈ S

, and select a control behavior strategy a = {a_(m), a_(f)},

a ∈ 𝒜

on the basis of a control object.

2) establish an optimal objective: according to a vehicle constraint relation at a safe merging moment t̂, construct an optimal objective space set

g* = {g₁^(*), g₂^(*), g₃^(*), g₄^(*), g₅^(*)}

for ramp merging,

g* ∈ 𝒢, 𝒢

being an objective space set,

g₁^(*)

representing that equation (1) that the merging vehicle is at the rear of the guidance vehicle is satisfied,

g₂^(*)

representing that equation (2) that the ramp merging vehicle is in front of the auxiliary vehicle is satisfied,

g₃^(*)

representing that equation (3) that the ramp merging vehicle and the guidance vehicle have a consistent speed is satisfied,

g₄^(*)

representing that equation (4) that the ramp merging vehicle and the auxiliary vehicle have a consistent speed is satisfied, and

g₅^(*)

representing that equation (5) that the ramp merging vehicle is safely merged from the acceleration lane into the main road in the selected merging area is satisfied;

3) construct an objective space: on the basis of classifications included by the optimal objective space set

g* = {g₁^(*), g₂^(*), g₃^(*), g₄^(*), g₅^(*))

in 2), establish the objective space set g to satisfy g = {g₁,g₂,g₃,g₄,g₅}, g₁ representing a position relation between the ramp merging vehicle and the guidance vehicle, g₂ representing a position relation between the ramp merging vehicle and the auxiliary vehicle, g₃ representing a speed relation between the ramp merging vehicle and the guidance vehicle, g₄ representing a speed relation between the ramp merging vehicle and the auxiliary vehicle, and g₅ representing a relation between a position of the ramp merging vehicle and the merging area; and

4) construct rewards: with a reward function beingr:

𝒮 × 𝒜 × 𝒢 → R

, introduce, according to safe, efficient and comfortable traveling requirements, at least two short-term objective rewards under the condition that the reward r(s, a, g*) at each time step includes a long-term objective merging reward QUOTE,

-   (1) the long-term objective merging reward R_(m)(t) that must be     included being represented as follows:, and -   $\begin{matrix}     \begin{array}{l}     {R_{m}(t) =} \\     \left\{ \begin{array}{l}     {\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu} 1,\mspace{6mu}\mspace{6mu}\mspace{6mu}\text{safe successful merging}} \\     {0,\mspace{6mu}\mspace{6mu}\mspace{6mu}\text{the merging vehicle can}\left( \text{may} \right)\text{be merged, but}g^{\text{*}}\mspace{6mu} is\mspace{6mu} not\mspace{6mu} satisfied} \\     {\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu} - 1\mspace{6mu},\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu}\text{others}}     \end{array} \right)     \end{array} & \text{­­­(7)}     \end{matrix}$ -   (2) other objectives being capable of including, but not limited to,     energy conservation, stability, comfort, efficiency, etc.     Construction may refer to:     -   a. construct a driving energy consumption reward R_(e)(t):         -   in consideration of battery efficiency and energy             consumption, use an ampere-time integration method to             construct a descent relation model of vehicle state of             charge (SOC):         -   $\begin{matrix}             {\Delta SOC(t)\mspace{6mu} = \mspace{6mu}\frac{\int_{0}^{t}{\left( {\text{V}_{oc} - \sqrt{V_{oc}^{2} - 4R_{int}\mspace{6mu} P_{b}(t)}} \right)dt}}{2R_{int}Q_{c}}} & \text{­­­(8)}             \end{matrix}$         -   V_(oc)being open circuit voltage, R_(int) being a             resistance, P_(b)(t) being battery power at a time t, and             Q_(c) being battery capacity.

Therefore, an energy-saving reward includes SOC situations of the ramp merging vehicle and the auxiliary vehicle as follows;

$\begin{matrix} {R_{e}(t)\mspace{6mu} = \mspace{6mu}\Delta SOC_{m}(t)\mspace{6mu} + \mspace{6mu}\Delta SOC_{f}(t)} & \text{­­­(9)} \end{matrix}$

b. construct a battery life reward R_(b)(t) according to

$\begin{matrix} {\Delta SOH(t)\mspace{6mu} = \mspace{6mu}\frac{1}{2N\left( \left| {P_{b}(t)} \right| \right)E_{0}(0)}{\int_{0}^{t}{\left| {P_{b}(t)} \right|dt}}} & \text{­­­(10)} \end{matrix}$

N being the number of battery cycles, and E₀(0) being standard battery capacity energy.

Therefore, the battery life reward is state of health (SOH) situations of the ramp merging vehicle and the auxiliary vehicle as follows: .

$\begin{matrix} {R_{b}(t)\mspace{6mu} = \,\Delta SOH_{m}\left( \text{t} \right)\mspace{6mu} + \mspace{6mu}\Delta SOH_{f}\left( \text{t} \right)} & \text{­­­(11)} \end{matrix}$

-   c. construct a stability reward R_(s)(t): without considering     lateral movement of the vehicle, see the stability reward as a     change of an actual acceleration, the maximum acceleration being     a_(max) = 3 m/s², and a specific model is as follows: . -   $\begin{matrix}     {R_{s}(t)\mspace{6mu} = \mspace{6mu}\frac{\left| {a_{f}(t)} \right|}{a_{\max}}} & \text{­­­(12)}     \end{matrix}$ -   d. construct a comfort reward construction R_(p)(t): in order to     reduce bump of the merging vehicles and improve comfort of     passengers, j_(max) representing a maximum allowable bump value,     m/s³ being taken as the unit, and the model is as follows: . -   $\begin{matrix}     {R_{p}(t)\mspace{6mu} = \mspace{6mu}\frac{\left| {\nabla a_{m}\left( \text{t} \right)} \right|\mspace{6mu} + \mspace{6mu}\left| {\nabla a_{f{(t)}}} \right|}{j_{\max}}} & \text{­­­(13)}     \end{matrix}$

A coupled Chebyshev based multi-objective reward optimization method specifically includes:

-   a. determine optimized reward terms: under the condition of assuming     that the number of a variety of introduced real-time short-term     objectives after successful merging is n_(r), determine the number     of the optimized reward terms for a merging problem as n_(r); -   b. determine a super-ideal optimal value of each reward: construct     the super-ideal optimal value -   r_(i)^(**) = r_(i)^(*) + ϑ_(i) (i = 1, 2, …, n_(r)) -   of each objective, -   r_(i)^(*) -   being an ideal value, and ϑ_(i) being a constant according to     selection of empirical data, and representing a degree that the     super-ideal optimal value is better than the ideal value; c.     construct a generalized weighted Chebyshev optimization model for a     multi-objective problem: with setting λ_(i) as a Chebyshev weight of     a short-term objective, transform the multi-objective problem into a     generalized weighted Chebyshev multi-objective problem as follows: -   $\min\left\{ {F_{c}\mspace{6mu} + \,\rho\mspace{6mu}{\sum_{i = 1}^{n_{r}}\left( {r_{i}{}^{**}\mspace{6mu} - \mspace{6mu} r_{i}} \right)}} \right\}\mspace{6mu}(14)$

The following condition being satisfied:

F_(c) − λ_(i)(r_(i)^(**) − r_(i)) ≥ 0   1 ≤ i ≤ n_(r)

r_(i) being an unrestricted variable 1 ≤ i ≤ n_(r) (15),

F_(c) ≥ 0

F_(c)being a weighted Chebyshev norm,

F_(c) = max {λ_(i)(r_(i)^(**) − r_(i))}

representing a maximum deviation between each objective and the super-ideal optimal value; and

$\rho\mspace{6mu}{\sum_{i = 1}^{n_{r}}\left( {r_{i}^{**}\mspace{6mu} - \mspace{6mu} r_{i}} \right)}$

being a term ensuring the stable algorithm, usually, ρ = 0.001, and a computation equation of λ_(i) being referred to as follows:

$\begin{matrix} {\lambda_{i}\mspace{6mu} = \mspace{6mu}\left\{ \begin{array}{ll} {\frac{1}{\left( {r_{i}^{\ast \ast}\mspace{6mu} - \mspace{6mu} r_{i}} \right)}\mspace{6mu}\left\lbrack {\sum_{i = 1}^{n_{r}}\frac{1}{\left( {r_{i}^{\ast \ast} - r_{i}} \right)}} \right\rbrack^{- 1}} & {r_{i}\text{­­­(16)} \neq \mspace{6mu} r_{i}^{\ast \ast}} \\ 1 & {r_{i}\mspace{6mu} = \mspace{6mu} r_{i}^{\ast \ast}} \\ 0 & {r_{i}\mspace{6mu} \neq \mspace{6mu} r_{i}^{\ast \ast},\mspace{6mu}\text{but}\mspace{6mu}\exists i,\mspace{6mu}\text{and}\mspace{6mu} r_{i}\mspace{6mu} = \, r_{i}^{\ast \ast}} \end{array} \right)} &  \end{matrix}$

c. construct a reward function as:

$\begin{matrix} {R_{m}(t)\mspace{6mu} = \mspace{6mu}\left\{ \begin{matrix} {1,\mspace{6mu}\text{safe}\mspace{6mu}\text{successful}\mspace{6mu}\text{merging}} \\ {0,\,\text{the}\mspace{6mu}\text{merging}\mspace{6mu}\text{vehicle}\,\text{can}\mspace{6mu}\left( \text{may} \right)\mspace{6mu}\text{be}\mspace{6mu}\text{merged,}\mspace{6mu}\text{but}g^{*}\mspace{6mu} is\mspace{6mu} not\mspace{6mu} satisfied.} \\ {- 1,\mspace{6mu}\text{others}} \end{matrix} \right)} & \text{­­­(17)} \end{matrix}$

5) obtain a data link: according to state, objective, strategy and reward data of the t-th time step obtained from 1) to 4), obtain the data link (s_(t)||g*, a, r, s_(t+1) ||g*), and store the data into an intelligent optimization module, s||g* representing a connection between a state s and an objective g*, and r not including the short-term objective R_(e)(t), R_(s)(t) when g* (Rm(t) being valued as 1, safe successful merging) is not completed. connection between a state s and an objective g*, and r not including the short-term objective R_(e)(t), R_(s)(t) when g* (Rm(t) being valued as 1, safe successful merging) is not completed.

6) expand data exploration, and optimize the objective space: according to 4) and 5), it may be found that data satisfying g* is difficult to obtain, because g* is a fixed final objective and has extremely limited guidance significance for the real-time rewards. Therefore, a multi-experience replay based virtual objective construction algorithm is further provided, to introduce virtual objectives, to optimize the objective space, and simultaneously expand the data exploration. The provided multi-experience replay based virtual objective construction algorithm includes:

-   (1) construct multi-experience virtual objectives and optimize the     objective space at a t time step: on the basis of the objective g*     and the data link (s_(t)||g*, a, r, s_(t+1)||g*) at the t time step,     construct a fully connected neural network H, g = H_(φ)(s, g*)     taking φ as a parameter, to obtain l virtual objectives in a current     state: g¹=Hφ(s_(t),g*), g²=H_(φ)(s_(t+1), g*), ...,     g^(l)=H_(φ)(s_(t+l-1),g*); -   (2) optimize data exploration on the basis of the optimized     objective space: construct l virtual objective data links on the     basis of the obtained l virtual objectives: -   (s_(t)∥g¹, a, r, s_(t + 1)∥g¹), (s_(t)∥g², a, r, s_(t + 1)∥g²), …, -   (s_(t)∥g^(l), a, r, s_(t + 1)∥g^(l)); and -   store the virtual objective data links into the intelligent     optimization module; -   (3) optimize data exploration at a full time step: repeat (1)     and (2) for each time step to complete data exploration and     objective space optimization at all time steps; -   (4) select virtual objective effectiveness on the basis of an     artificial intelligence model:     -   train the artificial intelligence model according to an         optimized data set, and select,     -   according to a training result, an optimal virtual objective         data link at each time step; and -   (5) verify the fully connected neural network H for the virtual     objectives: according to the optimal virtual objective data link at     each time step, verify the parameter φ of the fully connected neural     network H for the virtual objectives, to constantly improve accuracy     of generation of the virtual objectives, and improve a training     speed of the algorithm.

in each time step, according to the data link stored by the intelligent optimization module, on the basis of an actor-critic algorithm framework, train a merging control strategy by means of a deep neural network taking θ_(A) as a parameter, the strategy directly outputting an action to control acceleration magnitudes and states of the ramp merging vehicle and the auxiliary vehicle and objective inputs, strategy optimization aiming at to find an optimal behavior strategy a, to maximize a return expectation of a whole trip, which specifically includes:

-   (1) according to a Bellman function, construct an optimal value     function as:

-   $\begin{matrix}     {Q_{\pi}\left( {s,a,\, g} \right)\, = \,{\sum_{s^{\prime}\,,r}{p\left( {s^{\prime},r\left| {\left( {s,g} \right),a} \right)} \right)}}\mspace{6mu}\left\lbrack {r\mspace{6mu} + \mspace{6mu}\underset{a^{\prime}}{\gamma max}Q\left( {s^{\prime},a,g} \right)} \right\rbrack} & \text{­­­(18)}     \end{matrix}$

-   -   Q_(π) being a value function, γ being a loss factor, s, a, g         being a state, a behavior strategy, and an objective at a         current time step respectively, and s′, a′, g′ being a state, a         behavior strategy, and an objective at a next time step         respectively:

-   (2) according to (1), determine a standard time difference update     equation as:

-   $\begin{matrix}     {Q_{\pi}\left( {s,a,g} \right):\mspace{6mu} = \mspace{6mu} Q_{\pi}\left( {s,a,g} \right)\mspace{6mu} + \mspace{6mu}\zeta\left( {r\mspace{6mu} + \,\underset{a^{\prime}}{\gamma max}Q_{\pi}\left( {s_{t + 1},\mspace{6mu} a^{\prime},g^{\prime}} \right)\mspace{6mu} - \mspace{6mu} Q_{\pi}\left( {s,a,g} \right)} \right)} & \text{­­­(19)}     \end{matrix}$

-   -   ζ being a learning rate; and     -   utilize a critic network taking θ_(c) as a parameter to estimate         a Q value according to an update rule of equation (16), so as to         construct a loss function of the critic network as follows:     -   $\begin{matrix}         {J_{c}\left( \theta_{c} \right)\mspace{6mu} = \mspace{6mu}\left( {Q_{\pi}\left( {s,a,g\mspace{6mu}\left| {\mspace{6mu}\theta_{c}} \right)} \right)\mspace{6mu} - \mspace{6mu}\left( {r\mspace{6mu} + \mspace{6mu}\gamma Q_{\pi}\left( {s\prime\mspace{6mu},g\prime\mspace{6mu},\mspace{6mu}\pi\left( {s\prime\mspace{6mu},\mspace{6mu} g\prime\mspace{6mu}\left| {\mspace{6mu}\theta_{A}} \right)} \right)\mspace{6mu}\left| {\mspace{6mu}\theta_{c}} \right)} \right)} \right)} \right)^{2}} & \text{­­­(20)}         \end{matrix}$     -   θ_(c) being a critic network parameter;

-   (3) utilize a stochastic gradient descent algorithm to update a     critical network parameter by minimizing the loss function as     follows:

-   $\begin{matrix}     {\Delta\theta_{c}\mspace{6mu} = \mspace{6mu}\zeta\nabla_{\theta_{c}}L_{C}\left( \theta_{C} \right)} & \text{­­­(21)}     \end{matrix}$

-   $\begin{matrix}     {\Delta\theta_{A}\, = \,\zeta\nabla_{\zeta}Q\left( {s,a,g\,\left| {\mspace{6mu}\theta_{c}} \right)} \right)\nabla_{\theta_{A}}\pi\left( {s,g\,\left| \theta_{A} \right)} \right)} & \text{­­­(22)}     \end{matrix}$

-   -   ∇ representing a gradient; and

-   (4) output an optimal control strategy under the driving state by     means of forward delivery of a trained network:

-   $\begin{matrix}     {a\mspace{6mu} = \mspace{6mu}\pi\mspace{6mu}\left( {s,g\mspace{6mu}\left| {\mspace{6mu}\theta_{A}} \right)} \right)} & \text{­­­(23)}     \end{matrix}$

obtain the optimal control strategy, to achieve optimal control of the highway scenario oriented multi-objective optimization control method and system for cooperative ramp merging of connected vehicles on a highway by means of each module, thereby achieving efficient, safe, and energy-saving driving in the ramp area.

What is described above is merely the preferred example of the present invention, and is not intended to limit the present invention in any form. Any amendments or equivalent changes made according to the technical essence of the present invention still fall within the scope of the present invention. 

What is claimed is:
 1. A multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway, comprising: step 1, collecting state data of vehicles in a control area of the highway, and analyzing and processing the state data, the control area comprising, an intersection of a main road and a ramp, a merging area, and partial road segments of the main road, the ramp and an acceleration lane of the highway, a range of the control area being a communication range of a road side unit, the road side unit being arranged at the intersection of the main road and the ramp of the highway, and the merging area being a preselected area comprising the partial road segments of the acceleration lane and road segments of the main road parallel to the partial road segments of the acceleration lane; step 2, constructing a set of alternatives for a ramp merging vehicle, an auxiliary vehicle, and a guidance vehicle according to the state data of the vehicles in the control area; step 3, inputting the set of alternatives into an artificial intelligence based ramp merging multi-objective control model separately, and further determining selection of the auxiliary vehicle and the guidance vehicle by means of an optimal value strategy; step 4, according to the auxiliary vehicle, the guidance vehicle, and the merging vehicle that are selected, controlling and adjusting accelerations of the auxiliary vehicle and the guidance vehicle, to ensure safe merging of the ramp merging vehicle from the acceleration lane into the main road in the selected merging area; and step 5, collecting state data of the ramp merging vehicle and the auxiliary vehicle, of which the accelerations are adjusted, and returning to step 4 to adjust accelerations at a next moment.
 2. The multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway of claim 1, wherein in step 1, the state data of the vehicles comprises positions, speeds, powder battery states, and corresponding moment information of the vehicles in the control area.
 3. The multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway of claim 1, wherein in step 1, the analyzing and processing the state data comprises: analyzing the data, extracting features, and fusing information.
 4. The multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway according to claim 1, wherein in steps 2 and 3, the optimal value strategy selected for the ramp merging vehicle, the guidance vehicle, and the auxiliary vehicle is as follows: i) selecting the merging vehicle: setting a vehicle, of which a front bumper is closest to a ramp exit line, on the ramp as the merging vehicle, and obtaining state information of all the vehicles in the control area in first t time steps; ii) according to position information of all the vehicles, preliminary selecting each z main road vehicles in the rear of and in front of the merging vehicle as alternative vehicles of the auxiliary vehicle and the guidance vehicle on the basis of front and rear relations between main line vehicles and the merging vehicle; iii) in all the alternative vehicles, selecting two vehicles that are adjacent to each other as a group of guidance vehicle and auxiliary vehicle, to construct an alternative set AL of the guidance vehicle and the auxiliary vehicle of the merging vehicle; and iv) using a traversal method to substitute combinations in the alternative set AL into the artificial intelligence based ramp merging multi-objective control model separately, and determining the merging vehicle, the auxiliary vehicle, and the guidance vehicle that are finally selected on the basis of a value function Q_(π) of the model.
 5. The multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway according to claim 4, wherein the state information of all the vehicles in the control area in the first t time steps comprises speeds, positions, and accelerations.
 6. The multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway according to claim 4, wherein in step 4) of the optimal value strategy selected for the merging vehicle, the auxiliary vehicle, and the guidance vehicle, constructing an objective function and constraints of the artificial intelligence based ramp merging multi-objective control model successfully merged into a moment comprises: i) setting t̂ as a safe merging moment to construct a position and speed relation to be satisfied by the guidance vehicle, the auxiliary vehicle, and the ramp merging vehicle at the safe merging moment: $\begin{matrix} {x_{m}\left( \hat{t} \right) \leq x_{l}\left( \hat{t} \right) - \left( {L_{1} + s_{0}} \right) - \tau v_{m}\left( \hat{t} \right)} & \text{­­­(1)} \end{matrix}$ $\begin{matrix} {x_{m}\left( \hat{t} \right) \geq x_{f}\left( \hat{t} \right) + \left( {L_{1} + s_{0}} \right) + \tau v_{f}\left( \hat{t} \right)} & \text{­­­(2)} \end{matrix}$ $\begin{matrix} {v_{m}\left( \hat{t} \right) \leq v_{l}\left( \hat{t} \right)} & \text{­­­(3)} \end{matrix}$ $\begin{matrix} {v_{m}\left( \hat{t} \right) \geq v_{f}\left( \hat{t} \right)} & \text{­­­(4)} \end{matrix}$ $\begin{matrix} {x_{m}\left( \hat{t} \right) \in \left\lbrack {d_{min},d_{max}} \right\rbrack} & \text{­­­(5)} \end{matrix}$ x_(l), v_(l) and α_(l) representing a position, a speed, and an acceleration of the guidance vehicle; x_(f), v_(f) and α_(f) representing a position, a speed, and an acceleration of the auxiliary vehicle; x_(m), v_(m) and a_(m) representing a position, a speed, and an acceleration of the ramp merging vehicle; τ being a constant time interval, L₁ being a length of the vehicle, s₀ being a pause gap, d_(min) and d_(max)being a start point and an end point of the merging area respectively, and the merging area having a length of d_(max) - d_(min); and the equation sequentially representing, from top to bottom, that the ramp merging vehicle is at the rear of the guidance vehicle, the ramp merging vehicle is in front of the auxiliary vehicle, the ramp merging vehicle and the guidance vehicle have a consistent speed, the ramp merging vehicle and the auxiliary vehicle have a consistent speed, and the ramp merging vehicle is safely merged from the acceleration lane into the main road in the selected merging area; and ii) under the condition of satisfying 1), further constructing an objective function C comprising, but not limited to, objectives of driving comfort, vehicle energy consumption, traffic efficiency, etc.; $\begin{matrix} {minC\left( \hat{t} \right) = min{\int_{0}^{\hat{t}}\left( {c^{1}C_{1}\left( \hat{t} \right) + c^{2}C_{2}\left( \hat{t} \right) + \cdots + c^{n}C_{n}\left( \hat{t} \right)} \right)}} & \text{­­­(6)} \end{matrix}$ C_(n)(t̂) representing cost functions of different objectives, and c^(n) representing parameters.
 7. The multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway according to claim 4, wherein in step 4) of the optimal value strategy selected for the merging vehicle, the guidance vehicle and the auxiliary vehicle, the artificial intelligence based ramp merging multi-objective control model is solved by using a reinforcement learning actor-critic algorithm, which specifically comprises: i) establishing a state space S and a behavior space 𝒜 : selecting, according to the state data of the guidance vehicle, the auxiliary vehicle, and the ramp merging vehicle, six-dimensional state information s = {x_(l), x_(m), x_(f), v_(l), v_(m), v_(f)} to represent the most relevant influence factor in an environment, s ∈ S, and selecting a control behavior strategy a = {a_(m), a_(f)}, a ∈ A on the basis of a control object; ii) establishing an optimal objective: according to a vehicle constraint relation at a safe merging moment t̂, constructing an optimal objective space set g^(*) = {g₁^(*), g₂^(*), g₃^(*), g₄^(*), g₅^(*)} for ramp merging, g* ∈ 𝒢, 𝒢 being an objective space set, g₁^(*) representing that equation (1) that the merging vehicle is at the rear of the guidance vehicle is satisfied, g₂^(*) representing that equation (2) that the ramp merging vehicle is in front of the auxiliary vehicle is satisfied, g₃^(*) representing that equation (3) that the ramp merging vehicle and the guidance vehicle have a consistent speed is satisfied, g₄^(*) representing that equation (4) that the ramp merging vehicle and the auxiliary vehicle have a consistent speed is satisfied, and g₅^(*) representing that equation (5) that the ramp merging vehicle is safely merged from the acceleration lane into the main road in the selected merging area is satisfied; iii) constructing an objective space: on the basis of classifications included by the optimal objective space set g^(*) = {g₁^(*), g₂^(*), g₃^(*), g₄^(*), g₅^(*)}( in 2), establishing the objective space set 𝒢 to satisfy g = {g₁, g₂, g₃, g₄, g₅}, g₁ representing a position relation between the ramp merging vehicle and the guidance vehicle, g₂ representing a position relation between the ramp merging vehicle and the auxiliary vehicle, g₃ representing a speed relation between the ramp merging vehicle and the guidance vehicle, g₄ representing a speed relation between the ramp merging vehicle and the auxiliary vehicle, and g₅ representing a relation between a position of the ramp merging vehicle and the merging area; iv) constructing rewards: with a reward function being r : 𝒮 × 𝒜 × 𝒢 → R , introducing, according to safe, efficient and comfortable traveling requirements, at least two short-term objective rewards under the condition that the reward r(s,a,g*) at each time step includes a long-term objective merging reward R_(m)(t), the long-term objective merging reward R_(m)(t) that must be inc3luded being represented as follows: ; $\begin{matrix} \begin{array}{l} {R_{m}(t) =} \\ \left\{ {0,\mspace{6mu}\begin{array}{l} {1,\mspace{6mu}\text{safe successful merging}} \\ {\text{the merging vehicle can}\left( \text{may} \right)\text{be merged, but}g^{\ast}isnotsatisfied} \\ {- 1,\text{others}} \end{array}} \right) \end{array} & \text{­­­(7)} \end{matrix}$ v) obtaining a data link: according to state, objective, strategy and reward data of the t^(th) time step obtained from 1) to 4), obtaining the data link (s_(t)||g*,a,r,s_(t+1)||g*), and storing the data into an intelligent optimization module, s||g* representing a connection between a state s and an objective g*; vi) expanding data exploration, and optimizing the objective space: further providing a multi-experience replay based virtual objective construction algorithm, to introduce virtual objectives, to optimize the objective space, and simultaneously expand the data exploration; and vii) in each time step, according to the data link stored by the intelligent optimization module, on the basis of an actor-critic algorithm framework, training a merging control strategy by means of a deep neural network taking θ_(A) as a parameter, the strategy directly outputting an action to control acceleration magnitudes and states of the ramp merging vehicle and the auxiliary vehicle and objective inputs, strategy optimization aiming at to find an optimal behavior strategy a, to maximize a return expectation of a whole trip, and finally, the optimal control strategy being output by means of forward delivery of a trained network: a. = π (s,g | θ_(A)).
 8. The multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway according to claim 7, wherein the short-term objective rewards in construction of the rewards of the reinforcement learning actor-critic algorithm comprises, but not limited to, an energy-saving reward R_(e)(t), a comfort reward R_(p)(t), an efficient traffic reward R_(s)(t), and a battery state reward R_(b)(t); and a coupled Chebyshev based multi-objective reward optimization method for the reinforcement learning actor-critic algorithm comprises: (i) determining optimized reward terms: under the condition of assuming that the number of a variety of introduced real-time short-term objectives after successful merging isn_(r), determining the number of the optimized reward terms for a merging problem as n_(r); (ii) determining a super-ideal optimal value of each reward: constructing the super-ideal optimal value r_(i)^(**) = r_(i)^(*) + ϑ_(i)(i = 1, 2, …, n_(r)) of each objective, r_(i)^(*) being an ideal value, and ϑ_(i) being a constant according to selection of empirical data, and representing a degree that the super-ideal optimal value is better than the ideal value; (iii) constructing a generalized weighted Chebyshev optimization model for a multi-objective problem: with setting λ_(i) as a Chebyshev weight of a short-term objective, transforming the multi-objective problem into a generalized weighted Chebyshev multi-objective problem as follows: $\begin{matrix} {\min\left\{ {F_{c} + \rho{\sum_{i = 1}^{n_{r}}\left( {r_{i}^{\ast \ast} - r_{i}} \right)}} \right\}} & \text{­­­(14)} \end{matrix}$ The following condition being satisfied: $\begin{matrix} \begin{array}{l} {F_{c} - \lambda_{i}\left( {r_{i}^{\ast \ast} - r_{i}} \right) \geq 0\quad 1 \leq i \leq n_{r}} \\ {r_{i}\text{being an unrestricted variable}1 \leq i \leq n_{r}} \\ {F_{c} \geq 0} \end{array} & \text{­­­(15)} \end{matrix}$ F_(c) being a weighted Chebyshev norm, F_(c) = max {λ_(i)(r_(i)^(**) − r_(i))} representing a maximum deviation between each objective and the super-ideal optimal value; and $\rho{\sum_{i = 1}^{n_{r}}\left( {r_{i}^{\ast \ast} - r_{i}} \right)}$ being a term ensuring the stable algorithm, usually, ρ = 0.001, and a computation equation of λ_(i) being referred to as follows: $\begin{matrix} {\lambda_{i} = \left\{ \begin{array}{ll} {\frac{1}{\left( {r_{i}^{\ast \ast} - r_{i}} \right)}\left\lbrack {\sum_{i = 1}^{n_{r}}\frac{1}{\left( {r_{i}^{\ast \ast} - r_{i}} \right)}} \right\rbrack^{- 1}} & {r_{i} \neq r_{i}^{\ast \ast}} \\ 1 & {r_{i} = r_{i}^{\ast \ast}} \\ 0 & {r_{i} \neq r_{i}^{\ast \ast},\text{­­­(16)}\exists i,\text{and}r_{i} = r_{i}^{\ast \ast}} \end{array} \right)} &  \end{matrix}$ (iv) constructing a reward function as: $\begin{matrix} \begin{array}{l} {R_{m}(t) =} \\ \left\{ {0,\mspace{6mu}\begin{array}{l} {1,\mspace{6mu}\text{safe successful merging}} \\ {\text{the merging vehicle can}\left( \text{may} \right)\text{be merged, but}g^{\ast}isnotsatisfied} \\ {- 1,\text{others}} \end{array}} \right) \end{array} & \text{­­­(17)} \end{matrix}$ .
 9. The multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway according to claim 7, wherein the multi-experience replay based virtual objective construction algorithm provided in construction of the rewards of the reinforcement learning actor-critic algorithm comprises: (i) constructing multi-experience virtual objectives and optimizing the objective space at a t time step: on the basis of the objective g* and the data link (s_(t)||g*,a,r,s_(t+1)||g*) at the t time step, constructing a fully connected neural network H, g = H_(φ)(s, g*)taking φ as a parameter, to obtain 1 virtual objectives in a current state: g¹=H_(φ)(s_(t),g*), g²=Hφ(s_(t+1),g*), ..., g^(l)=H_(φ)(s_(t+l-1),g*); (ii) optimizing data exploration on the basis of the optimized objective space: constructing 1 virtual objective data links on the basis of the obtained 1 virtual objectives: (s_(t)||g¹,a,r,s_(t+1)||g¹), (s_(t)||g²,a,r,s_(t+1)||g²), ..., (s_(t)||g^(l),a,r,s_(t+1)||g^(l)); and storing the virtual objective data links into the intelligent optimization module; (iii) optimizing data exploration at a full time step: repeating ① and ② for each time step to complete data exploration and objective space optimization at all time steps; (iv) selecting virtual objective effectiveness on the basis of an artificial intelligence model: training the artificial intelligence model according to an optimized data set, and selecting, according to a training result, an optimal virtual objective data link at each time step; and (v) verifying the fully connected neural network H for the virtual objectives: according to the optimal virtual objective data link at each time step, verifying the parameter φ of the fully connected neural network H for the virtual objectives, to constantly improve accuracy of generation of the virtual objectives, so as to ensure algorithm performance and a training speed.
 10. A system based on the multi-objective optimization control method for cooperative ramp merging of connected vehicles on a highway of any one of claims 1-9, comprising: an information collection module, a data transmission module, a traffic control module, and an intelligent optimization module, wherein the information collection module is used for collecting state data of vehicles in a control area of a highway, analyzing and processing the state data, and selecting a ramp merging vehicle, an auxiliary vehicle, and a guidance vehicle; the information collection module comprises an in-vehicle unit, and a road side unit, the road side unit being arranged at an intersection of a main road and a ramp of a highway, the road side unit being used for collecting positions, speeds and corresponding moment information of the vehicles in the control area, and being further used for collecting time of determining the ramp merging vehicle, and time when a front bumper reaches a ramp exit line, and the in-vehicle unit being used for collecting power battery states and corresponding moment information of the vehicles in the control area; the data transmission module is used for using a mobile communication technology as a subject information transmission communication mode to assist one or two of wireless communication modes of wireless fidelity/bluetooth (WiFi/BT), and dedicated short range communications (DSRC) to transmit data between the information collection module and the traffic control module, the traffic control module and the intelligent optimization module; the traffic control module is used for obtaining, according to the state data of the vehicles provided by the information collection module, a behavior strategy a, an objective g and a reward r that are optimal in real time, sending the behavior strategy to the in-vehicle unit for real-time control of the vehicles, and further sending the behavior strategy a, the objective g and the reward r that are optimal to the intelligent optimization module; and the intelligent optimization module is used for storing data sent by the traffic control module, optimizing a ramp merging multi-objective control model on the basis of an alternative set AL of the vehicles selected in step 3) and an optimization algorithm provided in step 4), and transmitting the optimized model to the traffic control module. 